[MXNET-703] Update to TensorRT 5, ONNX IR 3. Fix inference bugs. #13310

KellenSunderland · 2018-11-17T00:52:32Z

Description

This PR updates the IR which is passed to TensorRT to use version3 of the spec, which aligns much better with MXNet defaults and results in a decrease in boilerplate code. This update also fixes some bugs when building inference engines that were resulting in feature vectors that were very different from what they should have been.

Fixes #12598
Fixes #13113

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Marco:
Resolves #13459

kalyc · 2018-11-19T18:38:28Z

@KellenSunderland please take a look at the failed CI build

@mxnet-label-bot add [pr-work-in-progress]

kalyc · 2018-11-19T18:40:29Z

@mxnet-label-bot add [pr-work-in-progress]

KellenSunderland · 2018-11-21T18:31:37Z

Note: will rebase this PR before merging.

larroy · 2018-11-29T23:04:22Z

src/operator/contrib/nnvm_to_onnx.cc

+  // More information on ONNX versions and opsets can be found at:
+  // https://github.com/onnx/onnx/blob/master/docs/IR.md
+
+  auto opset_proto = model_proto.add_opset_import();


is copy ok?

Yeah I believe so, proto editing in C++ is a little strange but I've seen this pattern in several places. I've run model validation and it certainly failed for me right away if I did not have a properly set opset proto associated with my model proto for onnx v3 models.

larroy

LGTM % comments

KellenSunderland · 2018-11-29T23:33:04Z

Thanks for the review @larroy! Will update.

@marcoabreu do you feel your comments were addressed sufficiently? Anything you'd still like to see changed?

marcoabreu · 2018-11-30T01:03:00Z

Lets wait until the Jenkins refactor has been finalized, ok?

NRauschmayr · 2018-12-08T08:18:36Z

I have two minor comments:

It would be good to have an environment variable to change the verbosity of nvinfer1::ILogger::Severity. The verbosity level kINFO gives details about the optimizations TensorRT has applied which can be useful for debugging purposes.
When I was investigating the MXNet TensorRT issue MXNet TensorRT generates wrong results #13113 , I was previsouly running on a GPU model that was not supported. It would be good if a warning would be given in this situation.

Roshrini · 2018-12-19T19:32:03Z

@marcoabreu Is jenking refactor finalized now? @KellenSunderland Can you rebase this PR?

Roshrini · 2019-01-02T19:42:33Z

@mxnet-label-bot Update [pr-awaiting-response]

This works around a CUDA 10 cmake issue documented here: clab/dynet#1457 This fix is temporary; once an updated cmake package is published to Ubuntu's package repo it may be reverted.

KellenSunderland · 2019-01-14T07:42:12Z

Rebased (sorry took a while), I want to do one more pass addressing Pedro, Marco and Nathalie's comments.

KellenSunderland · 2019-01-15T16:35:58Z

@NRauschmayr Totally agree with your comments here. Would it be ok if we make those changes in a follow up PR?

NRauschmayr · 2019-01-15T23:39:10Z

@KellenSunderland Sure, that's ok! There is no hurry for the changes which I suggested.

…che#13310) * [MXNET-703] Install CUDA 10 compatible cmake This works around a CUDA 10 cmake issue documented here: clab/dynet#1457 This fix is temporary; once an updated cmake package is published to Ubuntu's package repo it may be reverted. * [MXNET-703] Update to TensorRT 5 ONNX IR 3. Fix inference bugs. * [MXNET-703] Describe onnx opsets and major version

KellenSunderland requested a review from anirudh2290 as a code owner November 17, 2018 00:52

KellenSunderland force-pushed the master branch 3 times, most recently from 3a9b7e7 to f663c47 Compare November 17, 2018 01:21

KellenSunderland mentioned this pull request Nov 17, 2018

[MXNET-703] Update onnx-tensorrt #13169

Closed

5 tasks

KellenSunderland changed the title ~~WIP [MXNET-703] Update TensorRT ONNX IR to v3. Fix inference bugs.~~ WIP [MXNET-703] Update TensorRT to v5 ONNX IR to v3. Fix inference bugs. Nov 17, 2018

KellenSunderland force-pushed the master branch 2 times, most recently from e410dd8 to f6133bf Compare November 18, 2018 01:46

KellenSunderland requested review from marcoabreu and szha as code owners November 18, 2018 01:46

KellenSunderland force-pushed the master branch 9 times, most recently from 06a47fa to 6150f4a Compare November 18, 2018 16:53

marcoabreu added the pr-work-in-progress PR is still work in progress label Nov 19, 2018

KellenSunderland force-pushed the master branch 6 times, most recently from 22f738f to 2260f59 Compare November 20, 2018 04:40

KellenSunderland force-pushed the master branch from 2260f59 to 5558ab4 Compare November 21, 2018 19:46

larroy reviewed Nov 29, 2018

View reviewed changes

larroy approved these changes Nov 29, 2018

View reviewed changes

marcoabreu added pr-awaiting-response PR is reviewed and waiting for contributor to respond and removed pr-awaiting-review PR is waiting for code review labels Jan 2, 2019

[MXNET-703] Install CUDA 10 compatible cmake

2165f79

This works around a CUDA 10 cmake issue documented here: clab/dynet#1457 This fix is temporary; once an updated cmake package is published to Ubuntu's package repo it may be reverted.

KellenSunderland force-pushed the master branch from b2cf02f to a264afc Compare January 14, 2019 07:40

KellenSunderland added the pr-work-in-progress PR is still work in progress label Jan 14, 2019

KellenSunderland force-pushed the master branch 2 times, most recently from a1030a2 to d4d9ba3 Compare January 14, 2019 19:19

KellenSunderland and others added 2 commits January 14, 2019 14:47

[MXNET-703] Update to TensorRT 5 ONNX IR 3. Fix inference bugs.

5165d4b

[MXNET-703] Describe onnx opsets and major version

600bc1f

KellenSunderland force-pushed the master branch from d4d9ba3 to 600bc1f Compare January 14, 2019 22:48

marcoabreu approved these changes Jan 15, 2019

View reviewed changes

KellenSunderland mentioned this pull request Jan 15, 2019

[MXNET-857] Add initial NVTX profiler implementation #12328

Merged

5 tasks

KellenSunderland added pr-awaiting-review PR is waiting for code review and removed pr-awaiting-response PR is reviewed and waiting for contributor to respond pr-work-in-progress PR is still work in progress labels Jan 16, 2019

KellenSunderland mentioned this pull request Jan 16, 2019

[v1.4.x] [MXNET-703] Update to TensorRT 5, ONNX IR 3. Fix inference bugs. #13897

Merged

5 tasks

marcoabreu approved these changes Jan 16, 2019

View reviewed changes

KellenSunderland merged commit cc15d9a into apache:master Jan 16, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MXNET-703] Update to TensorRT 5, ONNX IR 3. Fix inference bugs. #13310

[MXNET-703] Update to TensorRT 5, ONNX IR 3. Fix inference bugs. #13310

KellenSunderland commented Nov 17, 2018 •

edited by marcoabreu

Loading

kalyc commented Nov 19, 2018

kalyc commented Nov 19, 2018

KellenSunderland commented Nov 21, 2018

larroy Nov 29, 2018

KellenSunderland Jan 14, 2019

larroy left a comment

KellenSunderland commented Nov 29, 2018

marcoabreu commented Nov 30, 2018

NRauschmayr commented Dec 8, 2018

Roshrini commented Dec 19, 2018

Roshrini commented Jan 2, 2019

KellenSunderland commented Jan 14, 2019

KellenSunderland commented Jan 15, 2019

NRauschmayr commented Jan 15, 2019

[MXNET-703] Update to TensorRT 5, ONNX IR 3. Fix inference bugs. #13310

[MXNET-703] Update to TensorRT 5, ONNX IR 3. Fix inference bugs. #13310

Conversation

KellenSunderland commented Nov 17, 2018 • edited by marcoabreu Loading

Description

Checklist

Essentials

kalyc commented Nov 19, 2018

kalyc commented Nov 19, 2018

KellenSunderland commented Nov 21, 2018

larroy Nov 29, 2018

Choose a reason for hiding this comment

KellenSunderland Jan 14, 2019

Choose a reason for hiding this comment

larroy left a comment

Choose a reason for hiding this comment

KellenSunderland commented Nov 29, 2018

marcoabreu commented Nov 30, 2018

NRauschmayr commented Dec 8, 2018

Roshrini commented Dec 19, 2018

Roshrini commented Jan 2, 2019

KellenSunderland commented Jan 14, 2019

KellenSunderland commented Jan 15, 2019

NRauschmayr commented Jan 15, 2019

KellenSunderland commented Nov 17, 2018 •

edited by marcoabreu

Loading